[opt](kerberos) use ticket cache instead of principal+keytab on BE side #47299

morningman · 2025-01-21T14:56:39Z

What problem does this PR solve?

Overview

Previously, BE node use principal and keytab to do the kerberos authentication.
But only the modified hadoop libhdfs support authenticating in this way, the origin libhdfs
only support setting kerberos ticket cache path, or use system level kerberos authentication context.

This pull request introduces a comprehensive Kerberos authentication module for the BE.
The module is designed to handle Kerberos ticket management, including initialization, authentication, and periodic ticket refresh.
It provides a robust interface for integrating Kerberos authentication, ensuring secure and efficient credential management.

Key Components

KerberosConfig (kerberos_config.h and kerberos_config.cpp):
- This class encapsulates the configuration settings required for Kerberos authentication, such as principal, keytab path, and refresh intervals.
- Provides methods to set and retrieve configuration parameters.
KerberosTicketCache (kerberos_ticket_cache.h and kerberos_ticket_cache.cpp):
- Manages the Kerberos ticket cache, including initialization, login, and periodic refresh of tickets.
- Supports operations like writing to the ticket cache and checking if a refresh is needed.
- Utilizes a background thread to periodically refresh tickets based on configured intervals.
- The default cache file will be written in /tmp dir, but can be modified using kerberos_ccache_path in be.conf
KerberosTicketMgr (kerberos_ticket_mgr.h and kerberos_ticket_mgr.cpp):
- Acts as a manager for multiple Kerberos ticket caches, handling their lifecycle, including creation, access, and cleanup.
- Provides methods to get or set ticket caches and retrieve cache file paths.
- Includes a background thread for cleaning up expired ticket caches every 1 hour. If a cache is longer being referenced, it will be removed.
HdfsMgr
- A simple and new class to manager the hdfs fs handler.
- It replace the old HdfsHandlerCache
- It will check HdfsHandler every 1 hour, and remove unused HdfsHandler after 24 hours.

Mainly Changes

Introduce a comprehensive kerberos ticket cache management on BE side
Use ticket cache path instead of principal and keytab to do the kerberos authentication of libhdfs.
Fix the issue that kerberos_krb5_conf_path in be.conf does not take effect.
Add a new system table backend_kerberos_ticket_cache, to view the krb ticket cache of each backend:

Doris > select * from information_schema.backend_kerberos_ticket_cache\G
*************************** 1. row ***************************
                  BE_ID: 1738304534666
                  BE_IP: 172.20.32.136
              PRINCIPAL: hdfs/[email protected]
                 KEYTAB: /path/to/hdfs.keytab
      SERVICE_PRINCIPAL: krbtgt/[email protected]
      TICKET_CACHE_PATH: /tmp/doris_krb_ce93d5ebb2a6554c7ba9f43aee3a9e6c
              HASH_CODE: ce93d5ebb2a6554c7ba9f43aee3a9e6c
             START_TIME: 2025-02-01 00:08:26
            EXPIRE_TIME: 2025-02-01 00:09:26
              AUTH_TIME: 2025-02-01 00:08:26
              REF_COUNT: 1
REFRESH_INTERVAL_SECOND: 3600

Usage

The user interface remains unchanged.

set krb5.conf path in be.conf kerberos_krb5_conf_path, default is /etc/krb5.conf
provide kerberos principal the keytab path as usual.

Configurations

be.conf

kerberos_ccache_path
The dir where kerber ticket cache file saved. the file name as format doris_krb_xxxx
kerberos_krb5_conf_path
The path of krb5.conf file
kerberos_refresh_interval_second
The min interval to refresh a kerberos ticket cache file. default is 1h.
cleanup logic

If the ticket cache is not used for 1 day, it will be deleted.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-01-21T14:56:45Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2025-01-27T17:38:09Z

run buildall

doris-robot · 2025-01-27T19:25:21Z

TPC-H: Total hot run time: 32324 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c80cf56ce658e77d06f21449423b647d0bbc2c5, data reload: false

------ Round 1 ----------------------------------
q1	17590	5654	5360	5360
q2	2046	325	177	177
q3	10464	1221	742	742
q4	10215	966	545	545
q5	7552	2371	2171	2171
q6	191	175	130	130
q7	882	759	622	622
q8	9221	1373	1236	1236
q9	5166	4981	4898	4898
q10	6833	2346	1863	1863
q11	482	276	266	266
q12	353	356	228	228
q13	17770	3699	3069	3069
q14	232	235	216	216
q15	534	476	476	476
q16	633	639	593	593
q17	589	874	338	338
q18	7066	6359	6493	6359
q19	1921	959	535	535
q20	321	324	186	186
q21	2840	2173	1990	1990
q22	367	352	324	324
Total cold run time: 103268 ms
Total hot run time: 32324 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5570	5508	5452	5452
q2	237	328	230	230
q3	2254	2651	2307	2307
q4	1394	1838	1338	1338
q5	4296	4738	4673	4673
q6	169	159	127	127
q7	2049	1951	1811	1811
q8	2617	2860	2722	2722
q9	7315	7293	7256	7256
q10	3020	3265	2780	2780
q11	591	514	498	498
q12	653	723	607	607
q13	3541	3958	3288	3288
q14	289	300	277	277
q15	507	472	459	459
q16	659	694	648	648
q17	1264	1750	1283	1283
q18	7747	7515	7279	7279
q19	826	952	1168	952
q20	2045	2051	1887	1887
q21	5832	5185	4909	4909
q22	601	595	554	554
Total cold run time: 53476 ms
Total hot run time: 51337 ms

doris-robot · 2025-01-27T19:37:48Z

TPC-DS: Total hot run time: 192037 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c80cf56ce658e77d06f21449423b647d0bbc2c5, data reload: false

query1	1310	949	951	949
query2	6316	2177	2093	2093
query3	10971	4488	4514	4488
query4	60430	29239	23098	23098
query5	5362	610	452	452
query6	440	193	179	179
query7	5514	503	300	300
query8	329	236	225	225
query9	8065	2733	2690	2690
query10	448	314	263	263
query11	17660	15096	15515	15096
query12	164	121	107	107
query13	1432	552	418	418
query14	11449	7119	7006	7006
query15	214	206	194	194
query16	6589	633	493	493
query17	1107	733	581	581
query18	1713	391	330	330
query19	237	192	168	168
query20	127	117	112	112
query21	211	132	105	105
query22	4673	4730	4517	4517
query23	34069	33565	33348	33348
query24	5572	2274	2337	2274
query25	463	462	411	411
query26	658	273	158	158
query27	1713	501	335	335
query28	4001	2543	2496	2496
query29	546	551	414	414
query30	212	189	157	157
query31	925	895	809	809
query32	65	64	62	62
query33	428	354	289	289
query34	756	867	526	526
query35	808	843	761	761
query36	1015	1026	980	980
query37	120	98	76	76
query38	4348	4337	4337	4337
query39	1502	1473	1431	1431
query40	205	112	97	97
query41	52	48	49	48
query42	135	104	106	104
query43	526	530	509	509
query44	1334	850	852	850
query45	184	184	161	161
query46	877	1054	658	658
query47	1964	1945	1878	1878
query48	391	429	326	326
query49	723	492	416	416
query50	676	676	392	392
query51	4310	4302	4229	4229
query52	115	106	93	93
query53	239	263	191	191
query54	489	532	429	429
query55	81	76	79	76
query56	260	268	268	268
query57	1176	1210	1132	1132
query58	245	237	235	235
query59	3341	3449	3185	3185
query60	284	273	251	251
query61	118	124	120	120
query62	731	713	640	640
query63	227	187	193	187
query64	1241	1008	682	682
query65	3243	3192	3202	3192
query66	735	394	300	300
query67	15984	15749	15347	15347
query68	5038	838	592	592
query69	486	302	263	263
query70	1214	1142	1076	1076
query71	401	283	258	258
query72	6051	3933	3825	3825
query73	798	744	359	359
query74	10071	8954	8827	8827
query75	3200	3131	2688	2688
query76	3796	1156	767	767
query77	490	347	269	269
query78	10138	9913	9343	9343
query79	2812	797	615	615
query80	1583	593	435	435
query81	554	274	231	231
query82	348	156	114	114
query83	270	173	148	148
query84	299	102	85	85
query85	772	363	344	344
query86	453	326	310	310
query87	4570	4505	4432	4432
query88	3977	2193	2121	2121
query89	387	314	293	293
query90	1583	193	192	192
query91	136	144	108	108
query92	66	57	57	57
query93	2849	855	536	536
query94	759	400	298	298
query95	338	274	260	260
query96	500	630	286	286
query97	2843	2902	2731	2731
query98	226	199	199	199
query99	1284	1338	1251	1251
Total cold run time: 310140 ms
Total hot run time: 192037 ms

doris-robot · 2025-01-27T19:43:10Z

ClickBench: Total hot run time: 31.23 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c80cf56ce658e77d06f21449423b647d0bbc2c5, data reload: false

query1	0.03	0.03	0.05
query2	0.07	0.04	0.03
query3	0.24	0.06	0.07
query4	1.64	0.11	0.11
query5	0.44	0.42	0.39
query6	1.14	0.67	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.52
query10	0.58	0.57	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.60
query14	2.83	2.84	2.82
query15	0.90	0.83	0.82
query16	0.38	0.39	0.38
query17	1.04	1.04	1.07
query18	0.22	0.21	0.21
query19	1.88	2.03	1.83
query20	0.02	0.01	0.01
query21	15.37	0.95	0.58
query22	0.76	0.74	0.66
query23	15.32	1.44	0.63
query24	3.49	1.44	1.65
query25	0.12	0.08	0.18
query26	0.36	0.15	0.14
query27	0.06	0.05	0.07
query28	14.51	0.90	0.42
query29	12.59	3.91	3.26
query30	0.26	0.09	0.07
query31	2.83	0.61	0.40
query32	3.22	0.55	0.46
query33	2.98	3.04	3.02
query34	16.50	5.15	4.46
query35	4.53	4.48	4.59
query36	0.65	0.52	0.49
query37	0.09	0.07	0.06
query38	0.04	0.03	0.03
query39	0.04	0.03	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.96 s
Total hot run time: 31.23 s

morningman · 2025-01-28T07:34:54Z

run buildall

doris-robot · 2025-01-28T08:14:12Z

TPC-H: Total hot run time: 31973 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f946e312640070cd561cb465feee2d6c4839e80b, data reload: false

------ Round 1 ----------------------------------
q1	17574	5401	5346	5346
q2	2041	307	169	169
q3	10448	1220	719	719
q4	10292	970	513	513
q5	8951	2349	2100	2100
q6	197	161	130	130
q7	888	743	611	611
q8	9236	1351	1172	1172
q9	5212	4889	4918	4889
q10	6816	2326	1886	1886
q11	478	277	257	257
q12	335	351	216	216
q13	18189	3755	3088	3088
q14	247	233	212	212
q15	508	481	463	463
q16	605	611	572	572
q17	567	862	328	328
q18	6872	6422	6319	6319
q19	2739	962	521	521
q20	300	317	185	185
q21	2760	2139	1965	1965
q22	361	335	312	312
Total cold run time: 105616 ms
Total hot run time: 31973 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5580	5456	5519	5456
q2	233	330	230	230
q3	2262	2628	2327	2327
q4	1390	1777	1366	1366
q5	4305	4690	4690	4690
q6	167	167	125	125
q7	2049	1969	1819	1819
q8	2649	2839	2674	2674
q9	7275	7255	7319	7255
q10	3023	3280	2806	2806
q11	568	513	482	482
q12	635	725	624	624
q13	3506	3855	3300	3300
q14	285	315	285	285
q15	525	497	480	480
q16	663	668	635	635
q17	1214	1757	1263	1263
q18	7546	7550	7252	7252
q19	814	1094	1083	1083
q20	1996	2037	1927	1927
q21	5740	5125	4858	4858
q22	600	581	568	568
Total cold run time: 53025 ms
Total hot run time: 51505 ms

doris-robot · 2025-01-28T08:26:40Z

TPC-DS: Total hot run time: 191165 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f946e312640070cd561cb465feee2d6c4839e80b, data reload: false

query1	1333	941	939	939
query2	6293	2099	2101	2099
query3	11061	4738	4456	4456
query4	61279	28722	22990	22990
query5	5645	569	424	424
query6	417	192	176	176
query7	5518	510	287	287
query8	319	218	232	218
query9	8373	2679	2661	2661
query10	467	308	248	248
query11	17300	15068	15603	15068
query12	158	109	104	104
query13	1436	546	439	439
query14	11602	6758	6599	6599
query15	201	211	193	193
query16	7334	672	519	519
query17	1177	745	588	588
query18	1768	408	307	307
query19	217	185	164	164
query20	118	116	115	115
query21	211	124	104	104
query22	4571	4789	4358	4358
query23	33988	33993	33420	33420
query24	5629	2342	2271	2271
query25	456	455	409	409
query26	663	254	155	155
query27	1764	472	320	320
query28	4388	2506	2486	2486
query29	557	530	427	427
query30	209	188	168	168
query31	932	893	815	815
query32	65	60	56	56
query33	433	358	297	297
query34	742	835	513	513
query35	791	838	799	799
query36	987	1009	954	954
query37	130	99	78	78
query38	4318	4307	4220	4220
query39	1543	1456	1446	1446
query40	196	115	106	106
query41	50	49	47	47
query42	126	106	103	103
query43	539	536	532	532
query44	1319	834	815	815
query45	186	174	174	174
query46	889	1056	649	649
query47	1938	1939	1866	1866
query48	386	415	321	321
query49	759	480	391	391
query50	639	694	400	400
query51	4297	4278	4293	4278
query52	113	105	97	97
query53	227	255	190	190
query54	486	514	423	423
query55	89	82	83	82
query56	268	269	245	245
query57	1195	1205	1152	1152
query58	245	244	244	244
query59	3239	3457	3316	3316
query60	273	261	241	241
query61	120	117	118	117
query62	721	773	644	644
query63	219	188	185	185
query64	1384	1095	642	642
query65	3235	3177	3131	3131
query66	765	400	287	287
query67	15917	15610	15505	15505
query68	5043	753	529	529
query69	478	288	252	252
query70	1169	1117	1134	1117
query71	407	293	266	266
query72	6078	3836	3737	3737
query73	769	756	358	358
query74	10283	8926	8850	8850
query75	3204	3127	2636	2636
query76	3798	1164	747	747
query77	482	389	285	285
query78	9935	9836	9227	9227
query79	3016	784	584	584
query80	674	523	458	458
query81	492	269	239	239
query82	355	150	114	114
query83	174	169	155	155
query84	289	91	73	73
query85	788	367	302	302
query86	376	316	295	295
query87	4422	4545	4521	4521
query88	4545	2166	2136	2136
query89	474	322	290	290
query90	1651	189	187	187
query91	136	134	106	106
query92	63	56	52	52
query93	2506	854	529	529
query94	730	376	281	281
query95	333	255	249	249
query96	500	658	286	286
query97	2784	2849	2786	2786
query98	225	201	195	195
query99	1313	1345	1250	1250
Total cold run time: 311705 ms
Total hot run time: 191165 ms

doris-robot · 2025-01-28T08:32:00Z

ClickBench: Total hot run time: 31.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f946e312640070cd561cb465feee2d6c4839e80b, data reload: false

query1	0.04	0.03	0.03
query2	0.10	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.43	0.41
query6	1.15	0.65	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.50	0.49
query10	0.56	0.55	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	2.81	2.87	2.75
query15	0.92	0.82	0.83
query16	0.40	0.40	0.37
query17	1.04	1.07	1.08
query18	0.23	0.21	0.20
query19	1.91	1.79	1.95
query20	0.01	0.01	0.01
query21	15.35	1.00	0.56
query22	0.76	0.88	0.65
query23	15.22	1.50	0.57
query24	3.60	1.44	1.52
query25	0.28	0.22	0.06
query26	0.20	0.14	0.14
query27	0.05	0.07	0.05
query28	14.16	1.00	0.43
query29	12.58	3.93	3.25
query30	0.25	0.09	0.07
query31	2.84	0.60	0.38
query32	3.22	0.56	0.46
query33	2.98	3.06	3.12
query34	16.45	5.32	4.50
query35	4.55	4.54	4.57
query36	0.66	0.48	0.47
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.63 s
Total hot run time: 31.13 s

doris-robot · 2025-01-28T08:35:40Z

TeamCity be ut coverage result:
Function Coverage: 42.09% (11005/26147)
Line Coverage: 32.37% (92981/287241)
Region Coverage: 31.51% (47651/151208)
Branch Coverage: 27.54% (24119/87572)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f946e312640070cd561cb465feee2d6c4839e80b_f946e312640070cd561cb465feee2d6c4839e80b/report/index.html

morningman · 2025-01-28T09:02:13Z

run buildall

doris-robot · 2025-01-28T10:50:06Z

TPC-H: Total hot run time: 32501 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7124069f5634503d3e44868c616e30f30661e1f7, data reload: false

------ Round 1 ----------------------------------
q1	17568	5519	5398	5398
q2	2053	324	193	193
q3	10373	1244	743	743
q4	10214	972	531	531
q5	7523	2457	2163	2163
q6	194	171	130	130
q7	901	771	603	603
q8	9216	1359	1182	1182
q9	5345	4930	4865	4865
q10	6856	2334	1919	1919
q11	477	275	256	256
q12	345	361	221	221
q13	17780	3815	3150	3150
q14	227	240	206	206
q15	518	488	447	447
q16	624	628	586	586
q17	576	901	335	335
q18	7110	6499	6493	6493
q19	1845	945	554	554
q20	314	345	195	195
q21	2908	2202	2019	2019
q22	361	337	312	312
Total cold run time: 103328 ms
Total hot run time: 32501 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5598	5482	5490	5482
q2	239	329	232	232
q3	2305	2679	2300	2300
q4	1374	1801	1351	1351
q5	4385	4786	4702	4702
q6	173	161	125	125
q7	2095	1993	1835	1835
q8	2668	2824	2760	2760
q9	7390	7240	7254	7240
q10	3051	3313	2782	2782
q11	574	507	478	478
q12	631	746	563	563
q13	3439	4013	3343	3343
q14	305	313	295	295
q15	512	481	461	461
q16	658	682	644	644
q17	1257	1748	1266	1266
q18	7874	7638	7367	7367
q19	826	1181	1094	1094
q20	2021	2081	1935	1935
q21	5882	5453	5060	5060
q22	605	595	578	578
Total cold run time: 53862 ms
Total hot run time: 51893 ms

doris-robot · 2025-01-28T11:02:03Z

TPC-DS: Total hot run time: 191616 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7124069f5634503d3e44868c616e30f30661e1f7, data reload: false

query1	1316	957	926	926
query2	6184	2142	2121	2121
query3	10967	4487	4586	4487
query4	32483	23890	23138	23138
query5	4351	613	457	457
query6	292	201	202	201
query7	3988	502	319	319
query8	295	270	231	231
query9	9621	2704	2710	2704
query10	473	323	242	242
query11	17779	15381	14862	14862
query12	156	110	102	102
query13	1563	538	401	401
query14	8716	7528	6395	6395
query15	272	213	192	192
query16	7994	670	521	521
query17	1610	740	579	579
query18	2115	412	308	308
query19	204	181	154	154
query20	124	116	130	116
query21	207	127	105	105
query22	4490	4462	4450	4450
query23	34487	33993	33321	33321
query24	6642	2378	2254	2254
query25	485	475	406	406
query26	820	276	158	158
query27	2344	474	334	334
query28	5192	2524	2515	2515
query29	670	572	431	431
query30	214	187	152	152
query31	966	880	833	833
query32	87	64	58	58
query33	479	350	329	329
query34	754	877	530	530
query35	821	871	747	747
query36	1006	1035	960	960
query37	123	101	88	88
query38	4452	4461	4294	4294
query39	1499	1409	1428	1409
query40	201	113	105	105
query41	49	50	47	47
query42	125	99	100	99
query43	530	543	507	507
query44	1327	835	844	835
query45	191	172	172	172
query46	881	1058	648	648
query47	1882	1888	1829	1829
query48	408	398	335	335
query49	745	484	401	401
query50	642	683	398	398
query51	4285	4295	4288	4288
query52	109	104	100	100
query53	241	282	200	200
query54	505	513	433	433
query55	91	81	80	80
query56	277	289	260	260
query57	1234	1207	1136	1136
query58	266	236	243	236
query59	3292	3306	3321	3306
query60	300	266	262	262
query61	123	120	120	120
query62	775	751	715	715
query63	228	194	198	194
query64	3624	1049	652	652
query65	3312	3235	3145	3145
query66	793	391	310	310
query67	15968	15561	15349	15349
query68	2075	820	558	558
query69	443	295	265	265
query70	1208	1135	1135	1135
query71	340	288	258	258
query72	6121	4037	3959	3959
query73	652	758	359	359
query74	10165	8971	8903	8903
query75	3182	3138	2658	2658
query76	2796	1169	745	745
query77	460	374	363	363
query78	10006	10014	9371	9371
query79	3187	809	584	584
query80	802	516	438	438
query81	515	278	240	240
query82	710	152	124	124
query83	174	174	155	155
query84	232	94	74	74
query85	770	350	301	301
query86	418	321	296	296
query87	4458	4540	4476	4476
query88	5103	2183	2171	2171
query89	390	332	285	285
query90	1805	187	184	184
query91	137	137	107	107
query92	64	60	52	52
query93	2714	880	540	540
query94	769	415	276	276
query95	339	261	252	252
query96	489	608	281	281
query97	2806	2849	2771	2771
query98	263	191	189	189
query99	1278	1368	1254	1254
Total cold run time: 283516 ms
Total hot run time: 191616 ms

doris-robot · 2025-01-28T11:07:22Z

ClickBench: Total hot run time: 30.51 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7124069f5634503d3e44868c616e30f30661e1f7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.07
query4	1.60	0.10	0.10
query5	0.43	0.42	0.40
query6	1.17	0.64	0.65
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.61	0.49	0.51
query10	0.56	0.57	0.54
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	2.72	2.72	2.83
query15	0.89	0.83	0.85
query16	0.39	0.38	0.36
query17	1.09	1.07	1.05
query18	0.23	0.21	0.20
query19	1.98	1.84	2.00
query20	0.01	0.01	0.02
query21	15.37	0.96	0.55
query22	0.74	0.99	0.56
query23	15.26	1.38	0.60
query24	2.84	1.72	1.17
query25	0.23	0.16	0.07
query26	0.25	0.14	0.14
query27	0.05	0.05	0.05
query28	13.98	0.94	0.43
query29	12.59	3.88	3.20
query30	0.24	0.09	0.06
query31	2.83	0.60	0.39
query32	3.23	0.55	0.45
query33	2.99	3.11	3.01
query34	16.47	5.15	4.43
query35	4.52	4.50	4.47
query36	0.65	0.51	0.49
query37	0.10	0.06	0.05
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.12
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.75 s
Total hot run time: 30.51 s

morningman · 2025-01-28T15:20:58Z

run buildall

doris-robot · 2025-01-28T16:01:03Z

TPC-H: Total hot run time: 32325 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b566aa4b3d3ef0ec297769ed599fa7f74e8f9164, data reload: false

------ Round 1 ----------------------------------
q1	17581	5431	5351	5351
q2	2045	301	167	167
q3	10439	1252	754	754
q4	10223	966	553	553
q5	8122	2447	2199	2199
q6	191	181	135	135
q7	905	769	610	610
q8	9223	1325	1120	1120
q9	5125	4920	4881	4881
q10	6942	2331	1872	1872
q11	470	277	255	255
q12	348	352	221	221
q13	17761	3712	3189	3189
q14	234	227	209	209
q15	505	467	477	467
q16	623	606	583	583
q17	577	869	326	326
q18	6852	6394	6503	6394
q19	1247	965	557	557
q20	318	324	192	192
q21	2793	2174	1987	1987
q22	370	333	303	303
Total cold run time: 102894 ms
Total hot run time: 32325 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5490	5534	5451	5451
q2	239	332	226	226
q3	2256	2797	2317	2317
q4	1392	1794	1374	1374
q5	4329	4709	4616	4616
q6	174	160	126	126
q7	2051	1931	1840	1840
q8	2629	2850	2704	2704
q9	7279	7182	7198	7182
q10	3037	3281	2809	2809
q11	592	535	499	499
q12	636	728	606	606
q13	3425	3854	3083	3083
q14	268	287	261	261
q15	506	460	479	460
q16	632	669	633	633
q17	1201	1739	1232	1232
q18	7167	7198	7100	7100
q19	778	1117	1003	1003
q20	1991	1942	1830	1830
q21	5480	5145	4963	4963
q22	591	610	560	560
Total cold run time: 52143 ms
Total hot run time: 50875 ms

doris-robot · 2025-01-28T16:13:03Z

TeamCity be ut coverage result:
Function Coverage: 42.09% (11005/26148)
Line Coverage: 32.36% (92944/287246)
Region Coverage: 31.51% (47652/151209)
Branch Coverage: 27.54% (24121/87572)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b566aa4b3d3ef0ec297769ed599fa7f74e8f9164_b566aa4b3d3ef0ec297769ed599fa7f74e8f9164/report/index.html

doris-robot · 2025-01-28T16:13:14Z

TPC-DS: Total hot run time: 196460 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b566aa4b3d3ef0ec297769ed599fa7f74e8f9164, data reload: false

query1	1337	957	926	926
query2	13109	9139	9298	9139
query3	6792	216	219	216
query4	33275	23216	22945	22945
query5	4344	655	460	460
query6	294	203	201	201
query7	4625	506	316	316
query8	303	252	235	235
query9	9509	2728	2683	2683
query10	441	339	265	265
query11	18325	15064	14953	14953
query12	150	110	101	101
query13	1635	534	412	412
query14	9685	7610	7662	7610
query15	222	197	201	197
query16	7205	672	469	469
query17	1541	739	567	567
query18	1968	385	312	312
query19	241	183	170	170
query20	141	117	115	115
query21	217	124	108	108
query22	4434	4629	4390	4390
query23	34057	33309	33509	33309
query24	6500	2398	2416	2398
query25	527	483	426	426
query26	748	288	162	162
query27	1888	483	337	337
query28	5173	2532	2523	2523
query29	622	615	460	460
query30	185	190	163	163
query31	1003	891	847	847
query32	88	64	64	64
query33	501	393	324	324
query34	778	907	513	513
query35	840	829	782	782
query36	1050	1037	973	973
query37	125	116	81	81
query38	4387	4421	4261	4261
query39	1493	1413	1457	1413
query40	169	118	104	104
query41	51	50	49	49
query42	120	101	108	101
query43	530	552	501	501
query44	1409	828	851	828
query45	182	174	175	174
query46	896	1066	672	672
query47	1899	1940	1817	1817
query48	394	419	329	329
query49	783	522	418	418
query50	678	700	418	418
query51	4287	4374	4261	4261
query52	106	103	102	102
query53	239	285	192	192
query54	495	526	410	410
query55	80	86	90	86
query56	278	275	275	275
query57	1179	1177	1156	1156
query58	247	256	243	243
query59	3448	3332	3328	3328
query60	306	281	280	280
query61	145	121	144	121
query62	790	715	669	669
query63	236	199	199	199
query64	3242	1041	679	679
query65	3315	3249	3257	3249
query66	900	396	321	321
query67	16698	16451	16109	16109
query68	6808	837	537	537
query69	477	303	262	262
query70	1200	1173	1147	1147
query71	434	288	257	257
query72	5155	3888	3845	3845
query73	668	760	366	366
query74	10267	9045	8841	8841
query75	4034	3147	2660	2660
query76	3539	1176	791	791
query77	760	409	276	276
query78	9967	9959	9318	9318
query79	3112	819	607	607
query80	739	545	439	439
query81	451	279	236	236
query82	686	155	124	124
query83	202	172	159	159
query84	278	90	74	74
query85	740	349	379	349
query86	366	315	311	311
query87	4568	4557	4464	4464
query88	3304	2222	2170	2170
query89	391	341	305	305
query90	1973	193	195	193
query91	136	143	112	112
query92	67	58	55	55
query93	1062	891	549	549
query94	665	410	305	305
query95	345	272	261	261
query96	491	616	292	292
query97	2786	2845	2758	2758
query98	227	195	205	195
query99	1492	1362	1285	1285
Total cold run time: 290250 ms
Total hot run time: 196460 ms

) ### What problem does this PR solve? Related PR: #47299 Problem Summary: The master branch only support java 17+, so should remote all `JAVA_OPTS` and use `JAVA_OPTS_FOR_JDK_17`

…de (apache#47299) ### What problem does this PR solve? #### Overview Previously, BE node use principal and keytab to do the kerberos authentication. But only the modified hadoop libhdfs support authenticating in this way, the origin libhdfs only support setting kerberos ticket cache path, or use system level kerberos authentication context. This pull request introduces a comprehensive Kerberos authentication module for the BE. The module is designed to handle Kerberos ticket management, including initialization, authentication, and periodic ticket refresh. It provides a robust interface for integrating Kerberos authentication, ensuring secure and efficient credential management. #### Key Components 1. **KerberosConfig** (`kerberos_config.h` and `kerberos_config.cpp`): - This class encapsulates the configuration settings required for Kerberos authentication, such as principal, keytab path, and refresh intervals. - Provides methods to set and retrieve configuration parameters. 2. **KerberosTicketCache** (`kerberos_ticket_cache.h` and `kerberos_ticket_cache.cpp`): - Manages the Kerberos ticket cache, including initialization, login, and periodic refresh of tickets. - Supports operations like writing to the ticket cache and checking if a refresh is needed. - Utilizes a background thread to periodically refresh tickets based on configured intervals. - The default cache file will be written in `/tmp` dir, but can be modified using `kerberos_ccache_path` in be.conf 3. **KerberosTicketMgr** (`kerberos_ticket_mgr.h` and `kerberos_ticket_mgr.cpp`): - Acts as a manager for multiple Kerberos ticket caches, handling their lifecycle, including creation, access, and cleanup. - Provides methods to get or set ticket caches and retrieve cache file paths. - Includes a background thread for cleaning up expired ticket caches every 1 hour. If a cache is longer being referenced, it will be removed. 4. **HdfsMgr** - A simple and new class to manager the hdfs fs handler. - It replace the old `HdfsHandlerCache` - It will check HdfsHandler every 1 hour, and remove unused HdfsHandler after 24 hours. #### Mainly Changes 1. Introduce a comprehensive kerberos ticket cache management on BE side 1. Use ticket cache path instead of principal and keytab to do the kerberos authentication of libhdfs. 2. Fix the issue that `kerberos_krb5_conf_path` in be.conf does not take effect. 3. Add a new system table `backend_kerberos_ticket_cache`, to view the krb ticket cache of each backend: ``` Doris > select * from information_schema.backend_kerberos_ticket_cache\G *************************** 1. row *************************** BE_ID: 1738304534666 BE_IP: 172.20.32.136 PRINCIPAL: hdfs/[email protected] KEYTAB: /path/to/hdfs.keytab SERVICE_PRINCIPAL: krbtgt/[email protected] TICKET_CACHE_PATH: /tmp/doris_krb_ce93d5ebb2a6554c7ba9f43aee3a9e6c HASH_CODE: ce93d5ebb2a6554c7ba9f43aee3a9e6c START_TIME: 2025-02-01 00:08:26 EXPIRE_TIME: 2025-02-01 00:09:26 AUTH_TIME: 2025-02-01 00:08:26 REF_COUNT: 1 REFRESH_INTERVAL_SECOND: 3600 ``` #### Usage The user interface remains unchanged. 1. set krb5.conf path in be.conf `kerberos_krb5_conf_path`, default is `/etc/krb5.conf` 2. provide kerberos principal the keytab path as usual. #### Configurations be.conf 1. `kerberos_ccache_path` The dir where kerber ticket cache file saved. the file name as format `doris_krb_xxxx` 2. `kerberos_krb5_conf_path` The path of krb5.conf file 6. `kerberos_refresh_interval_second` The min interval to refresh a kerberos ticket cache file. default is 1h. 7. cleanup logic If the ticket cache is not used for 1 day, it will be deleted.

### What problem does this PR solve? add more hudi jni test cases. These tests depends on apache#47192 and part of apache#47299(the jni-util part)

…che#47913) ### What problem does this PR solve? Related PR: apache#47299 Problem Summary: The master branch only support java 17+, so should remote all `JAVA_OPTS` and use `JAVA_OPTS_FOR_JDK_17`

### What problem does this PR solve? add more hudi jni test cases. These tests depends on apache#47192 and part of apache#47299(the jni-util part)

### What problem does this PR solve? Introduced from #47299 the `SetEnvIfNecessary` may be called multiple times, and in #47299, we changed `setenv("LIBHDFS_OPTS", libhdfs_opts.c_str(), 0);` to `setenv("LIBHDFS_OPTS", libhdfs_opts.c_str(), 1);` so it will add `krb5 path` at the end of `LIBHDFS_OPTS` and set it every time, so `LIBHDFS_OPTS` becomes longer and longer. This PR fix this issue by calling `SetEnvIfNecessary` only once

…kerberos ticket. (#47826) ### What problem does this PR solve? Related PR: #47299 Problem Summary: fix the `KerberosTicketEntry entry` initialization to enable compile pass.

…48352) pick part of #47299 when calling `hdfsBuilderSetKerb5Conf`, the `value` string's lifecycle must be with `hdfs_builder`.

…48353) pick part of #47299 when calling `hdfsBuilderSetKerb5Conf`, the `value` string's lifecycle must be with `hdfs_builder`.

### What problem does this PR solve? add more hudi jni test cases. These tests depends on #47192 and part of #47299(the jni-util part)

…VA_OPTS #48170 #48335 (#48284) bp #48170 and #48335 and also pick part of #47299, only related to `jni-util.cpp`. To pass the krb5.conf config to Jni env

### What problem does this PR solve? Introduced from apache#47299 the `SetEnvIfNecessary` may be called multiple times, and in apache#47299, we changed `setenv("LIBHDFS_OPTS", libhdfs_opts.c_str(), 0);` to `setenv("LIBHDFS_OPTS", libhdfs_opts.c_str(), 1);` so it will add `krb5 path` at the end of `LIBHDFS_OPTS` and set it every time, so `LIBHDFS_OPTS` becomes longer and longer. This PR fix this issue by calling `SetEnvIfNecessary` only once

…kerberos ticket. (apache#47826) ### What problem does this PR solve? Related PR: apache#47299 Problem Summary: fix the `KerberosTicketEntry entry` initialization to enable compile pass.

### What problem does this PR solve? Introduced from apache#47299 the `SetEnvIfNecessary` may be called multiple times, and in apache#47299, we changed `setenv("LIBHDFS_OPTS", libhdfs_opts.c_str(), 0);` to `setenv("LIBHDFS_OPTS", libhdfs_opts.c_str(), 1);` so it will add `krb5 path` at the end of `LIBHDFS_OPTS` and set it every time, so `LIBHDFS_OPTS` becomes longer and longer. This PR fix this issue by calling `SetEnvIfNecessary` only once

…kerberos ticket. (apache#47826) ### What problem does this PR solve? Related PR: apache#47299 Problem Summary: fix the `KerberosTicketEntry entry` initialization to enable compile pass.

…stead of using kerberos ticket cache. (#48655) ### What problem does this PR solve? Related PR: #47299, #49181 This PR mainly changes: 1. Back to use principal and keytab to login kerberos instead of using kerberos ticket cache. Discard what I did in #47299. It looks like there are a lot of issue when using ticket cache in multi-kerberos env. So I abandoned that logic. 2. Config's default value Change the default value of related to hdfs file handle cache 1. `max_hdfs_file_handle_cache_num`: from 1000 to 20000 2. `max_hdfs_file_handle_cache_time_sec`: from 3600 to 28800 3. Fix a bug the cleanup thread of `FileHandleCache` is not working

morningman force-pushed the be_ticket_path branch 4 times, most recently from 972fb7e to 8c80cf5 Compare January 27, 2025 17:37

morningman changed the title ~~[opt](kerberos) use ticket patch instead of keytab~~ [opt](kerberos) use ticket cache instead of keytab on BE side Jan 28, 2025

morningman closed this Jan 28, 2025

morningman reopened this Jan 28, 2025

morningman marked this pull request as ready for review January 28, 2025 09:31

morningman requested review from gavinchou and dataroaring as code owners January 28, 2025 09:31

morningman added dev/2.1.x dev/3.0.x labels Jan 28, 2025

morningman mentioned this pull request Feb 14, 2025

[fix](java) should use JAVA_OPTS_FOR_JDK_17 instead of JAVA_OPTS #47913

Merged

16 tasks

morningman added the dev/2.1.x-lh label Feb 17, 2025

morningman mentioned this pull request Feb 25, 2025

branch-3.0: [fix](java) should use JAVA_OPTS_FOR_JDK_17 instead of JAVA_OPTS #48170 #48335 #48284

Merged

suxiaogang223 mentioned this pull request Feb 25, 2025

[cherry-pick](branch-3.0) add more hudi jni test cases (#47431) #48311

Merged

16 tasks

morningman mentioned this pull request Feb 25, 2025

[fix](jvm) the jvm opt should only be set once #48335

Merged

16 tasks

morningman added a commit that referenced this pull request Feb 26, 2025

branch-2.1:[fix](libhdfs) fix the lifecycle issue of libhdfs config (#…

82fe8bc

…48352) pick part of #47299 when calling `hdfsBuilderSetKerb5Conf`, the `value` string's lifecycle must be with `hdfs_builder`.

dataroaring pushed a commit that referenced this pull request Feb 26, 2025

branch-3.0:[fix](libhdfs) fix the lifecycle issue of libhdfs config (#…

e81a403

…48353) pick part of #47299 when calling `hdfsBuilderSetKerb5Conf`, the `value` string's lifecycle must be with `hdfs_builder`.

dataroaring added the dev/3.0.5-merged label Feb 26, 2025

dataroaring pushed a commit that referenced this pull request Feb 26, 2025

[cherry-pick](branch-3.0) add more hudi jni test cases (#47431) (#48311)

c88f97e

### What problem does this PR solve? add more hudi jni test cases. These tests depends on #47192 and part of #47299(the jni-util part)

morningman removed the dev/3.0.5-merged label Feb 26, 2025

morningman mentioned this pull request Mar 4, 2025

[fix](kerberos) Back to use principal and keytab to login kerberos instead of using kerberos ticket cache. #48655

Merged

16 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[opt](kerberos) use ticket cache instead of principal+keytab on BE side #47299

[opt](kerberos) use ticket cache instead of principal+keytab on BE side #47299

morningman commented Jan 21, 2025 •

edited

Loading

hello-stephen commented Jan 21, 2025

morningman commented Jan 27, 2025

doris-robot commented Jan 27, 2025

doris-robot commented Jan 27, 2025

doris-robot commented Jan 27, 2025

morningman commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

morningman commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

morningman commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

[opt](kerberos) use ticket cache instead of principal+keytab on BE side #47299

[opt](kerberos) use ticket cache instead of principal+keytab on BE side #47299

Conversation

morningman commented Jan 21, 2025 • edited Loading

What problem does this PR solve?

Overview

Key Components

Mainly Changes

Usage

Configurations

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

hello-stephen commented Jan 21, 2025

morningman commented Jan 27, 2025

doris-robot commented Jan 27, 2025

doris-robot commented Jan 27, 2025

doris-robot commented Jan 27, 2025

morningman commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

morningman commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

morningman commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

doris-robot commented Jan 28, 2025

morningman commented Jan 21, 2025 •

edited

Loading